Three Dimensional User Interface Effects On A Display

ABSTRACT

The techniques disclosed herein may use various sensors to infer a frame of reference for a hand-held device. In fact, with various inertial clues from accelerometer, pyrometer, and other instruments that report their states in real time, it is possible to track a Frenet frame of the device in real time to provide an instantaneous (or continuous) 3D frame-of-reference. In addition to—or in place of—calculating this instantaneous (or continuous) frame of reference, the position of a user&#39;s head may either be inferred or calculated directly by using one or more of a device&#39;s optical sensors, e.g., an optical camera, infrared camera, laser, etc. With knowledge of the 3D frame-of-reference for the display and/or knowledge of the position of the user&#39;s head, more realistic virtual 3D depictions of the graphical objects on the device&#39;s display may be created—and interacted with—by the user.

BACKGROUND

It's no secret that video games now use various properties of motion andposition collected from, e.g., compasses, accelerometers, gyrometers,and Global Positioning System (GPS) units in hand-held devices orcontrol instruments to improve the experience of play in simulated,i.e., virtual, three dimensional (3D) environments. In fact, software toextract so-called “six axis” positional information from such controlinstruments is well-understood, and is used in many video games today.The first three of the six axes describe the “yaw-pitch-roll” of thedevice in three dimensional space. In mathematics, the tangent, normal,and binormal unit vectors for a particle moving along a continuous,differentiable curve in three dimensional space are often called T, N,and B vectors, or, collectively, the “Frenet frame,” and are defined asfollows: T is the unit vector tangent to the curve, pointing in thedirection of motion; N is the derivative of T with respect to thearclength parameter of the curve, divided by its length; and B is thecross product of T and N. The “yaw-pitch-roll” of the device may also berepresented as the angular deltas between successive Frenet frames of adevice as it moves through space. The other three axes of the six axesdescribe the “X-Y-Z” position of the device in relative threedimensional space, which may also be used in further simulatinginteraction with a virtual 3D environment.

Face detection software is also well-understood in the art and isapplied in many practical applications today including: digitalphotography, digital videography, video gaming, biometrics,surveillance, and even energy conservation. Popular face detectionalgorithms include the Viola-Jones object detection framework and theSchneiderman & Kanade method. Face detection software may be used inconjunction with a device having a front-facing camera (or other opticalsensor(s)) to determine when there is a human user present in front ofthe device, as well as to track the movement of such a user in front ofthe device.

However, current systems do not take into account the location andposition of the device on which the virtual 3D environment is beingrendered in addition to the location and position of the user of thedevice, as well as the physical and lighting properties of the user'senvironment in order to render a more interesting and visually pleasinginteractive virtual 3D environment on the device's display.

Thus, there is need for techniques for tracking the movement of anelectronic device having a display, as well as the lighting conditionsin the environment of a user of such an electronic device and themovement of the user of such an electronic device—and especially theposition of the user of the device's eyes and/or head. With informationregarding lighting conditions in the user's environment, the position ofthe user's eyes and/or head, and an instantaneous (or continuous) 3Dframe-of-reference for the display of the electronic device, virtual 3Ddepictions of the objects on the device's display may be created thatare more appropriately drawn for the user's current point of view withrespect to the device's display.

SUMMARY

The techniques disclosed herein use various position sensors, e.g., acompass, a Micro-Electro-Mechanical Systems (MEMS) accelerometer, a GPSmodule, and a MEMS gyrometer, to infer a 3D frame of reference (whichmay be a non-inertial frame of reference) for a personal electronicdevice, e.g., a hand-held device such as a mobile phone. Use of theseposition sensors can provide a true Frenet frame for the device, i.e.,X- and Y- vectors for the display, and also a Z-vector that pointsperpendicularly to the display. In fact, with various inertial cluesfrom an accelerometer, gyrometer, and other instruments that reporttheir states in real time, it is possible to track the Frenet frame ofthe device in real time, thus providing an instantaneous (or continuous)3D frame of reference for the hand-held device. Once an instantaneous(or continuous) frame of reference of the device is known, thetechniques that will be disclosed herein can then either infer theposition of the user's eyes and/or head, or calculate the position ofthe user's eyes and/or head directly by using a front-facing camera orother optical sensor(s). With an instantaneous (or continuous) 3Dframe-of-reference for the display and/or the position of the user'seyes and/or head, more realistic virtual 3D depictions of graphicalobjects on the device's display may be created and interacted with.

To accomplish a more realistic virtual 3D depiction of the objects onthe device's display, objects may be rendered on the display as if theywere in a real 3D “place” in the device's operating system environment.In some embodiments, the positions of objects on the display can becalculated by ray tracing their virtual coordinates, i.e., theircoordinates in the virtual 3D world of objects, back to the user of thedevice's eyes and/or head and intersecting the coordinates of theobjects with the real plane of the device's display. In otherembodiments, virtual 3D user interface (UI) effects, referred to hereinas “2½D” effects, may be applied to 2D objects on the device's displayin response to the movement of the device, the movement of the user(e.g., the user's head), or the lighting conditions in the user'senvironment in order to cause the 2D objects to “appear” to be virtuallythree dimensional to the user.

3D UI Effects Achievable Using this Technique

It is possible, for instance, using a 2½D depiction of a user interfaceenvironment to place realistic moving shines or moving shadows on thegraphical user interface objects, e.g., icons, displayed on the devicein response to the movement of the device, the movement of the user, orthe lighting conditions in the user's environment.

It is also possible to create a “virtual 3D operating systemenvironment” and allow the user of a device to “look around” a graphicaluser interface object located in the virtual 3D operating systemenvironment in order to see its “sides.” If the frame of reference ismagnified to allow the user to focus on a particular graphical userinterface object, it is also possible for the user to rotate the objectto “see behind” it as well, via particular positional changes of thedevice or the user, as well as user interaction with the device'sdisplay.

It is also possible to render the virtual 3D operating systemenvironment as having a recessed “bento box” form factor inside thedisplay. Such a form factor would be advantageous for modularinterfaces. As the user rotated the device, he or she could look intoeach “cubby hole” of the bento box independently. It would also then bepossible, via the use of a front-facing camera or other opticalsensor(s), to have visual “spotlight” effects follow the user's gaze,i.e., by having the spotlight effect “shine” on the place in the displaythat the user is currently looking into. It is also possible to controla position of a spotlight effect based solely on a determined 3D frameof reference for the device. For example, the spotlight effect could beconfigured to shine into the cubby hole whose distorted normal vectorpointed the closest in direction to the user of the device's currentposition.

Interaction with a Virtual 3D World Inside the Display

To interact via touch with the virtual 3D display, the techniquesdisclosed herein make it possible to ray trace the location of the touchpoint on the device's display into the virtual 3D operating systemenvironment and intersect the region of the touch point with whateverobject or objects it hits. Motion of the objects caused by touchinteraction with the virtual 3D operating system environment could occursimilarly to how it would in a 2D mode, but the techniques disclosedherein would make it possible to simulate collision effects and otherphysical manifestations of reality within the virtual 3D operatingsystem environment. Further, it is possible to better account for issuessuch touchscreen parallax, i.e., the misregistration between the touchpoint and the intended touch location being displayed, when the Frenetframe of the device is known.

How to Prevent the GPU from Constantly Re-Rendering

To prevent the over-use of the Graphics Processing Unit (GPU) andexcessive battery drain on the device, the techniques disclosed hereinemploy the use of a particularized gesture to turn on the “virtual 3Doperating system environment” mode, as well as positional quiescence toturn the mode off. In one embodiment, the gesture is the so-called“princess wave,” i.e., the wave-motion rotation of the device about oneof its axes. For example, the “virtual 3D operating system environment”mode can be turned on when more than three waves of 10-20 degrees alongone axis occur within the span of one second.

In one embodiment, when the “virtual 3D operating system environment”mode turns on, the display of the UI “unfreezes” and turns into a 3Ddepiction of the operating system environment (preferably similar to the2D depiction, along with shading and textures indicative of 3D objectappearance). When the mode is turned off, the display could slowlytransition back to a standard orientation and freeze back into the 2D or2½D depiction of the user interface environment. Positional quiescence,e.g., holding the device relatively still for two to three seconds,could be one potential cue to the device to freeze back to the 2D or 2½Doperating system environment mode and restore the display of objects totheir more traditional 2D representations.

Desktop Machines as Well

On desktop machines, the Frenet frame of the device doesn't change, andthe position of the user with respect to the device's display wouldlikely change very little, but the position of the user's eyes and/orhead could change significantly. A front-facing camera or other opticalsensor(s), in conjunction with face detection software, would allow theposition of the user's eyes and/or head to be computed. Usingfield-of-view information for the camera, it would also be possible toestimate the distance of the user's head from the display, e.g., bymeasuring the head's size or by measuring the detected user'spupil-to-pupil distance and assuming a canonical measurement for thehuman head, according to ergonomic guidelines. Using this data, it wouldthen be possible to depict a realistic 2½D or 3D operating systemenvironment mode, e.g., through putting shines on windows, title bars,and other UI objects, as well as having them move in response to themotion of the user's eyes or the changing position of the user's head.Further, it would also be possible to use the position of the user'shead and eyes to allow the user to “look under” a window after the usershifts his or her head to the side and/or moves his or her head towardsthe display.

Because of innovations presented by the embodiments disclosed herein,the 3D UI effects utilizing properties of motion that are describedbelow may be implemented directly by a personal electronic device'shardware and/or software, making the techniques readily applicable toany number of personal electronic devices, such as mobile phones,personal data assistants (PDAs), portable music players, televisions,gaming consoles, portable gaming devices, as well as laptop, desktop,and tablet computers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary personal electronic device.

FIG. 2 illustrates an exemplary 3D UI technique that may be employed bya personal electronic device operating in a 2½D operating systemenvironment mode.

FIG. 3 illustrates a personal electronic device presenting a virtual 3Ddepiction of a graphical user interface object lit by a virtual lightsource, in accordance with one embodiment.

FIG. 4 illustrates a personal electronic device presenting a virtual 3Ddepiction of a graphical user interface object lit by a representativeambient light source, in accordance with one embodiment.

FIG. 5 illustrates the Frenet frame for a personal electronic device, inaccordance with one embodiment.

FIG. 6 illustrates the effects of device movement on a personalelectronic device presenting a virtual 3D depiction of a graphical userinterface object, in accordance with one embodiment.

FIG. 7 illustrates the effects of user movement on a personal electronicdevice presenting a virtual 3D depiction of a graphical user interfaceobject, in accordance with one embodiment.

FIG. 8 illustrates a recessed, “bento box” form factor inside thevirtual 3D display of a personal electronic device, in accordance withone embodiment.

FIG. 9 illustrates a point of contact with the touchscreen of a personalelectronic device ray traced into a virtual 3D operating systemenvironment, in accordance with one embodiment.

FIG. 10 illustrates an exemplary gesture for activating the display of apersonal electronic device to operate in a virtual 3D operating systemenvironment mode, in accordance with one embodiment.

FIG. 11 illustrates, in flowchart form, one embodiment of a process foroperating a personal electronic device in a virtual 3D operating systemenvironment mode.

FIG. 12 illustrates, in flowchart form, one embodiment of a process fortoggling a personal electronic device between operating in a virtual 3Doperating system environment mode and a non-virtual 3D operating systemenvironment mode.

FIG. 13 illustrates, in flowchart form, one embodiment of a process forprojecting spotlights indicative of the position of a user's eyes into avirtual 3D operating system environment of a personal electronic device.

FIG. 14 illustrates, in flowchart form, one embodiment of a process forimplementing graphical user interface effects on the display of apersonal electronic device based on ambient light sources detected inthe environment of the device and/or the relative position of thedevice.

FIG. 15 illustrates a simplified functional block diagram of a personalelectronic device, in accordance with one embodiment.

DETAILED DESCRIPTION

This disclosure pertains to techniques for tracking the movement of anelectronic device having a display, as well as lighting conditions inthe environment of a user of such an electronic device and the movementof the user of such an electronic device—and especially the position ofthe user of the device's eyes and/or head. With the position of theuser's eyes and/or head and an instantaneous (or continuous) 3Dframe-of-reference for the display of the electronic device, morerealistic virtual 3D depictions of the objects on the device's display(i.e., depictions that are more appropriately drawn for the user'scurrent point of view with respect to the device's display) may becreated and interacted with. While this disclosure discusses a newtechnique for creating more realistic virtual 3D depictions of theobjects on a personal electronic device's display, one of ordinary skillin the art would recognize that the techniques disclosed may also beapplied to other contexts and applications as well. The techniquesdisclosed herein are applicable to any number of electronic devices withpositional sensors, proximity sensors, and/or optical sensors: such asdigital cameras, digital video cameras, mobile phones, personal dataassistants (PDAs), portable music players, televisions, gaming consoles,portable gaming devices, desktop, laptop, and tablet computers, and gamecontrollers. An embedded processor, such a Cortex A8 with the ARM® v7-Aarchitecture, provides a versatile and robust programmable controldevice that may be utilized for carrying out the disclosed techniques.(CORTEX® and ARM® are registered trademarks of the ARM Limited Companyof the United Kingdom.)

In the interest of clarity, not all features of an actual implementationare described in this specification. It will of course be appreciatedthat in the development of any such actual implementation (as in anydevelopment project), numerous decisions must be made to achieve thedevelopers' specific goals (e.g., compliance with system- andbusiness-related constraints), and that these goals will vary from oneimplementation to another. It will be appreciated that such developmenteffort might be complex and time-consuming, but would nevertheless be aroutine undertaking for those of ordinary skill having the benefit ofthis disclosure. Moreover, the language used in this disclosure has beenprincipally selected for readability and instructional purposes, and maynot have been selected to delineate or circumscribe the inventivesubject matter, resort to the claims being necessary to determine suchinventive subject matter. Reference in the specification to “oneembodiment” or to “an embodiment” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least one embodiment of the invention, andmultiple references to “one embodiment” or “an embodiment” should not beunderstood as necessarily all referring to the same embodiment.

Referring now to FIG. 1, an exemplary personal electronic device 100 isshown. Personal electronic device 100 in FIG. 1 is depicted as a mobilephone, but this is not the only type of device in which the techniquesdisclosed herein may be implemented, e.g., the device may also comprisea computer monitor, watch, music player, television screen, dashboarddisplay in a car or other vehicle, etc., wherein it is beneficial toprovide 3D effects on a device that are more appropriately drawn for theuser's current point of view with respect to the device's display.Device 100 is depicted as having display 102, which may be a capacitivetouchscreen interface capable of displaying graphical objects andreceiving touch input from a user. Device 100 is also depicted as havingone or more optical sensors 112 and a proximity sensor 114, which maycomprise, e.g., an infrared sensor. Optical sensor(s) 112 may comprise,e.g., a two-dimensional camera(s), a stereoscopic camera(s), an infraredcamera(s), and/or a laser(s). The optical sensors(s) may be used tolocate a user of device 100 and estimate a distance of the user to thedisplay 102 of device 100, in addition to the position and direction ofgaze of the user's eyes and/or the position of the user's head—as willbe explained further below. Proximity sensor 114 may be one of a numberof well-known proximity sensors known and used in the art, and it may beused, e.g., to detect the presence of nearby objects without anyphysical contact. As is known in the art, a proximity sensor often emitsan electromagnetic or electrostatic field, or a beam of electromagneticradiation. Moreover, non-optical means may also be used to obtainequivalent information, such as eyewear or headgear worn by a user,e.g., caps and hats capable of providing position information. Further,external means, such as detectors available in the user's surroundingenvironment, e.g., detectors in a car dashboard, mounted on computermonitors, installed in room, etc., may also be used to obtain equivalentinformation. Optical sensor(s) 112 and proximity sensor 114 may also beused to measure light levels in the environment around the user and tolocate light sources in the user's environment. As described below, suchinformation may be useful in making a realistic virtual 3D depiction ofa graphical user interface that appears to be “responsive” to lightsources located in the “real world.” Display colors may also be modifiedaccording to the measured color of the ambient light in the user'senvironment. In FIG. 1, display 102 is shown to be displaying severalgraphical user interface objects, e.g., icon 104. Icon 104 may beindicative of a program, file, or other application that the device iscapable of executing should the icon be selected by a user. Also shownon display 102 is an object springboard 110. In the prior art, icons 106have been depicted as sitting on the surface of springboard 110 to givethem the appearance of being in a 3D operating system environment.Additional visual cues may be added to the icons 104/106, such as shines113 and/or reflections 108 to further enhance the 3D appearance of theicons 104/106. In some embodiments, this enhanced 2D representation maybe referred to herein as 2½D.

Referring now to FIG. 2, an exemplary 3D UI technique that may beemployed by a personal electronic device operating in a 2½D operatingsystem environment mode is illustrated, in accordance with oneembodiment. In FIG. 2, icon 202 is presented in three different virtuallighting environments, 208 a-c. In each environment, shadow layer 200 isrepositioned appropriately underneath icon 202 to create the illusionthat virtual light source 204 is causing a realistic shadow to be castby icon 202. For example, in virtual lighting environment 208 a, virtuallighting source 204 a is positioned directly over icon 202 a. Thus,shadow layer 200 is placed directly below icon 202 a, i.e., with nooffset. This is indicated by coordinate 206 a, representing the lowerleft corner of shadow layer 200 in virtual lighting environment 208 a.Hypothetical coordinates (x, y) have been assigned to coordinate 206 ain order to illustrate the repositioning of shadow layer 200 in virtuallighting environments 208 b and 208 c as compared to its position invirtual lighting environment 208 a.

In virtual lighting environment 208 b, virtual lighting source 204 b hasbeen positioned above and to the left of icon 202 b. Thus, shadow layer200 is placed below icon 202 b and offset slightly to the right. Theamount of offset, s, is determined based on the position and distancebetween the virtual lighting source and the icon, and creates arealistic lighting effect on icon 202 b. The offset, s, is indicated bycoordinate 206 b, representing the lower left corner of shadow layer 200in virtual lighting environment 208 b and having hypotheticalcoordinates (x+s, y).

In virtual lighting environment 208 c, virtual lighting source 204 c hasbeen positioned above and to the right of icon 202 c. Thus, shadow layer200 is placed below icon 202 c and offset slightly to the left. Theamount of offset, s, is determined based on the position and distancebetween the virtual lighting source and the icon, and creates arealistic lighting effect on icon 202 c. The offset, s, is indicated bycoordinate 206 c, representing the lower left corner of shadow layer 200in virtual lighting environment 208 c and having hypotheticalcoordinates (x−s, y).

The location of virtual lighting source 204 in a virtual lightingenvironment used by a personal electronic device employing thetechniques disclosed herein may be, e.g., determined programmatically bythe device's operating system, selected manually by the user of thedevice, or positioned by the device's operating system to simulate thelocation of an ambient light source detected in the user's environment.That is, if a bright light source is located directly above the user ofa device, the device's operating system may place the virtual lightingsource directly above the virtual lighting environment, so as toaccurately simulate the lighting conditions in the user's environment.Further, multiple light sources may be detected and simulated in thevirtual lighting environment, and the intensities of such detected lightsources may also be accurately simulated.

The 2½D operating system environment techniques, e.g., the one describedabove in reference to FIG. 2, may be rendered in a relativelycomputationally cheap and flexible manner by utilizing a suitablegraphical programming framework such as Apple Inca's CORE ANIMATION®framework. (CORE ANIMATION® is a registered trademark of Apple Inc.) Forexample, shadow layer 200 is an example of a separate graphical layerthat may simply be repositioned at the desired location in the virtuallighting environment to create the illusion that the virtual lightingsource has moved. One advantage of the repositioning technique describedherein is that the pixels comprising the graphical layer do not need tobe re-rendered each time the device's operating system determines that arepositioning of the graphical layer is necessitated due to e.g., themovement of the device or a change in the lighting conditions in theuser's environment. Another type of graphical layer, known as a “shinemap” may also be implemented as a separate graphical layer and can berepositioned over an icon or other graphical user interface object tocreate the illusion of different virtual lighting scenarios. Again, oneadvantage to this technique is that the icon or other graphical userinterface object content underneath the shine map does not have to bere-rendered each time the device's operating system determines that arepositioning of the graphical layer is necessitated.

Referring now to FIG. 3, a personal electronic device 100 presenting avirtual 3D depiction 306 of a graphical user interface object 106 lit bya simulated, i.e., virtual, light source 304 is illustrated, inaccordance with one embodiment. FIG. 3 also shows a side view of virtual3D operating system environment 300. The two dimensional icons 104/106depicted in FIG. 1 have been replaced with virtual 3D representations105/107 of the same icon objects within virtual 3D operating systemenvironment 300. As shown in FIG. 3, icon object is still sitting onspringboard 110. As shown by axes 320, in FIG. 3, the X-axis extendsalong the width of the display 102 and into the page, the Y-axis extendsalong the length of display 102, and the Z-axis extends perpendicularlyaway from the surface of display 102. The additional three columns oficons depicted on display 102 in FIG. 1 are not visible as they wouldextend along the X-axis into the page in the view presented in FIG. 3.It is also to be understood that, in some embodiments, each and everygraphical user interface object in the virtual 3D operating systemenvironment 300, including graphics as well as photographs and otherimages, could be depicted on display 102 of device 100 with appropriate3D effects. This application focuses only on the representation of asingle graphical user interface object, icon 107, for simplicity andclarity.

As is known in the 3D graphics arts, virtual light source 304 may becreated by the device's processor(s) and “placed” at various spots inthe virtual 3D operating system environment 300 in order to generaterealistic shading and shine effects on objects displayed by devicedisplay 102. As illustrated in FIG. 3, light source 304 is placed at thetop and center of virtual 3D operating system environment 300, althoughsuch placement is not strictly necessary. Dashed line 318 indicates thepath from virtual graphical user interface object 107 to light source304. Utilizing the path of dashed line 318 will become important laterwhen certain aspects of ray tracing are discussed in conjunction withrendering a realistic 3D depiction of user interface objects on devicedisplay 102.

The eyes (and, by extension, the head) of the user of device 100 arerepresented by element 302 in FIG. 3. Dashed line 308 indicates the pathfrom the user's eyes (and/or head) to the optical sensor 112 of device100. As mentioned above, optical sensor 112 may be used to: estimate thelength of dashed line 318 (that is, the distance of the user's eyesand/or head to the device); detect (and potentially recognize) the faceof the user; measure the current distance between the user's pupils; orto locate light sources in the user's environment. Additionally,proximity sensor 114 may be used to further gauge ambient light levelsin the user's environment. Further, light sources in the user'senvironment may also be detected by measuring specular reflections offthe pupils of the user's eyes.

Vector 310 represents a binormal vector that points perpendicularly outfrom the display 102. Vector 312 represents a vector extending from thedisplay 102 directly to the position of the center of user's eyes and/orhead 302 along the X-axis. Dashed line 314 is depicted to helpillustrate the position of user's eyes and/or head 302 in the directionof the X-axis with respect to the display 102. The angle 316 betweenvector 310 and vector 312 can be used by device 100 to determine theangle at which the user is likely viewing display 102. As illustrated inFIG. 3, the user appears to be slightly to the left of the binormalvector 310, i.e., slightly closer to the “surface” of the page ratherthan deeper “into” the page. The current position of the user in FIG. 3is also reflected in the depiction of graphical user interface object306 as having a small amount of its left side 307 visible on thedevice's display 102 for the user's current orientation and positionshown in FIG. 3. As the user's eyes and/or head 302 move farther andfarther to the left with respect to the display 102 of device 100, i.e.,as angle 316 increases, the user will be able to see more and more ofthe left side 307 of graphical user interface object 306. One way ofunderstanding the 3D effect resulting from the application of suchtechniques is to envision the display screen 102 of the hand-held device100 as a “window” into 3D space 300. As with windows in the real word,as an observer gets closer to or farther from the window, or looksthrough the window at different angles, the observer is able to seedifferent objects, different angles of objects, etc. To achieve theseeffects, a technique known as ray tracing may be utilized.

Ray tracing, as known in the art, involves simulating the path a photonwould take moving from a light source, to an object an observer isviewing, and then reflected to the observer's eye—only in reverse order.Thus, this process is also sometimes referred to as “backward raytracing.” In one embodiment of a ray tracing process, for each pixel indisplay 102, a ray 304 is extended from the observer's eye, through thegiven pixel, and into the virtual 3D operating system environment 300.Whatever virtual 3D object is “pierced” by the projection of the rayinto the virtual 3D environment is then displayed in the correspondingpixel in display 102. This process results in the creation of a view onthe display screen that is the same view an observer would get if he orshe was looking through a window into the virtual 3D environment.

A further step may be implemented once a virtual 3D object has been“pierced” by the projection of the ray into the virtual 3D environment.Specifically, a second ray 318 may be extended from the virtual 3Dobject to the position of the virtual lighting source 304. If anotherobject is pierced by the second ray 318 as it is extended to the lightsource 304, it is a cue to the graphics rendering engine to place ashadow on that pixel. In the case of FIG. 3, it appears that ray 318intersects icon objects 105 and 109, indicating that icon object 107would have a shadow cast upon its top surface if the user were to adjusthis or her own position (or the position of the device) such that he orshe could see the top surface of icon object 107. If no object isdetected between the pixel being rendered and the light source, then therelative distance to and intensity of the light source may be used tocalculate the brightness intensity of the given pixel as it is displayedon device display 102. Utilizing these techniques can produce virtual 3Drenderings of environments that accurately portray the effects thatlighting would have on the virtual 3D objects if they were actually in areal world setting.

The techniques described above in relation to FIG. 3 allow the GPU ofthe device to apply appropriate lighting and perspective transformationsto the virtual 3D depiction of at least one graphical user interfaceobject on the display of the device, thus resulting in a more realisticand immersive virtual 3D experience for the user of the device.According to some embodiments, the above-described lighting andperspective transformations used to produce the virtual 3D renderings onthe display may be turned off if more than one user is detected withinviewing proximity of the device.

Referring now to FIG. 4, a personal electronic device 100 presenting avirtual 3D depiction 406 of a graphical user interface object 107 lit bya representative ambient light source 400 is illustrated, in accordancewith one embodiment. As in the embodiment described above with respectto FIG. 3, ray tracing techniques may be employed by the GPU of device100 in order to present a realistic virtual 3D depiction of virtual 3Doperating system environment 300. In FIG. 4, however, the lightingsource for the virtual 3D operating system environment 300 is not avirtual light source created arbitrarily by the operating system of thedevice; rather, the virtual 3D operating system environment 300 isdepicted as though it were being lit by real world ambient light source400. Thus, the ray traced back from the virtual 3D operating systemenvironment to ambient light source 400 for object 107 is represented bydashed line 408. Device 100's optical sensor(s) 112 or proximity sensor114 may be utilized to determine the distance to (represented by dashedline 402)—and relative position of—the brightest real world ambientlight source 400, as well as the intensity of real world ambient lightsource 400. Once this information has been ascertained, the GPU maylight the objects in virtual 3D operating system environment 300according to well-known lighting and shading techniques in the 3Dgraphics arts.

As shown in FIG. 4, ambient light source 400 is slightly to the right ofthe device's display 102, i.e., slightly farther “into” the page alongthe X-axis. The current position of the ambient light source 400 in FIG.3 is also reflected in the depiction of graphical user interface object406 as having a small amount of shading 407 on its left side visible onthe device's display 102 for the ambient light source 400's currentorientation and position as shown in FIG. 4. As the ambient light source400 moves farther and farther to the right with respect to the display102 of device 100, i.e., as the ambient light source 400 moves deeperand deeper into the page along the X-axis, there would be more and moreshading 404 created on the left sides of the graphical user interfaceobjects 105/107 in the virtual 3D operating system environment 300. Ofcourse, such changes in shading on graphical user interface objects inthe virtual 3D operating system environment would likewise be reflectedin the rendering of the objects on the device's display 102 should thedevice 100 and/or user be positioned and oriented such that a shadedside of a graphical user interface object is visible from the vantagepoint of the user's eyes 302.

In addition to using these techniques, the operating system software ofthe device can determine some light source positions in the user'senvironment by analyzing the front-facing camera's (or other opticalsensor(s)) view. Specifically, the device may attempt to locate thebrightest areas in the front-facing camera's (or other opticalsensor(s)) view. To avoid falsely identifying specular reflections(e.g., reflections of a pair of eyeglasses) as light sources, the imagereceived from the front-facing camera (or other optical sensor(s)) maybe blurred using a small radius, such as with a Gaussian blur at astandard deviation of 3.0, thus reducing the salience of small specularreflections. In one embodiment, the brightest 5% of the image, ifbrighter than a predetermined brightness value, may be recognized as alight source and located by thresholding the image, a technique which isknown in the image processing art. Once thresholded, the shape i.e., thealpha mask, as well as the centroid of the light source may be computed.The computed centroid may be used to determine the light source'sbearing direction in real-world 3D space. Further, aluminance-thresholded version of the front-facing camera's (or otheroptical sensor(s)) image may be used to put realistic reflections on therenderings of curved objects in the virtual 3D operating systemenvironment. Additionally, if there are no sufficiently bright ambientlights sources located in the environment of the user, the process maydefault to using a virtual light source, as in FIG. 3 above.

It is also possible, for usage of a device outdoors, to compute theposition of the Sun as a real-world light source in the user'senvironment. This can be done by using Greenwich Mean Time (GMT), theGPS location of the device, and compass bearings, as well as the Frenetframe of the device in the local geographic coordinate system. Thetechnique for the computation of the Sun's relative position at a knownpoint on the earth is well-known. Once computed, the location of the Sunmay be utilized to generate a virtual light source for the virtual 3Doperating system environment that is representative of the Sun's currentlocation.

Referring now to FIG. 5, the Frenet frame 500 for a personal electronicdevice 100 is illustrated, in accordance with one embodiment. Asdiscussed above, in mathematics, a Frenet Frame is defined by thetangent, normal, and binormal unit vectors for a particle moving along acontinuous, differentiable curve in three dimensional space, which areoften called T, N, and B vectors, as shown in FIG. 5. The T is the unitvector tangent to the curve, pointing in the direction of motion; N isthe derivative of T with respect to the arclength parameter of thecurve, divided by its length; and B is the cross product of T and N. Asshown in FIG. 5, T is aligned with the Y axis of the display, N isaligned with the X axis of the display, and B is normal to the plane ofthe display. Together, the T, N, and B vectors of the Frenet frame tiethe display coordinate system to the coordinate system of the realworld.

By utilizing the various positional sensors in device 100, such as acompass, a Micro-Electro-Mechanical Systems (MEMS) accelerometer, a GPSmodule, and a MEMS gyrometer, a 3D frame of reference may be inferredfor the device 100. In fact, it is now possible to track the Frenetframe of the device in real time, thus providing an instantaneous (orcontinuous) 3D frame of reference for the hand-held device. Once theframe of reference of the device is known, in addition to the positionof the user's eyes and/or head, more realistic virtual 3D depictions ofthe objects on the device's display may be created and interacted with,as is explained further with respect to FIG. 6 below.

Referring now to FIG. 6, the effects of device movement 600 on apersonal electronic device 100 presenting a virtual 3D depiction 606 ofa graphical user interface object 107 is illustrated, in accordance withone embodiment. Notice that the Frenet frame 608 of the device 100 asoriented in FIG. 6 is substantially different from the Frenet frame 500of the device 100 as oriented in FIG. 5. Specifically, the device 100has been tilted such that the eyes 302 of the user look slightlydownwards into virtual 3D operating system environment 602. Arrow 600 isindicative of the difference in position of device 100 between FIG. 5and FIG. 6. As can be seen from the projection ray traces 604 extendingfrom the eyes 302 of the user through the plane of the device display102 and into the virtual 3D operating system environment 602, thedepiction 607 of graphical user interface object 107 shows a smallamount of graphical user interface object 107's top surface 607 on thedevice's display 102. This is in contrast to the depiction 406 ofgraphical user interface object 107 shown in FIG. 4, wherein the“window” into the virtual 3D operating system environment 300 (i.e., the“window” being the display 102 of device 100) is more parallel withrespect to the orientation of the vertical stacks of icon objects in thevirtual 3D operating system environment, and, thus, the user is unableto see the top surface of graphical user interface object 107. Applyingthese techniques, it becomes clear that the user would be able tomanipulate the position of the device and/or his or her eyes and/or headin order to see the side surfaces of graphical user interface objects oreven behind graphical user interface objects.

Referring now to FIG. 7, the effects of user movement on a personalelectronic device 100 presenting a virtual 3D depiction 700 of agraphical user interface object 107 is illustrated, in accordance withone embodiment. As in FIG. 3, dashed line 712 indicates the path fromthe user's eyes to the optical sensor(s) 112 of device 100. As mentionedabove, optical sensor(s) 112 may be used to: estimate the length ofdashed line 712 (that is, the distance of the user's eyes and/or head tothe device); detect (and potentially recognize) the face of the user;measure the current distance between the user's pupils; or to locatelight sources in the user's environment. Vector 706 represents abinormal vector that points perpendicularly out from the display 102.Vector 708 represents a vector extending from the display 102 directlyto the position of the center of user's eyes 702 along the X-axis.Dashed line 710 is depicted to help illustrate the position of user'seyes 702 in the direction of the X-axis with respect to the display 102.The angle 704 between vector 706 and vector 708 can be used by device100 to determine the angle at which the user is likely viewing display102. As illustrated in FIG. 7, the user appears to be slightly to theleft of the binormal vector 706, i.e., slightly closer to the “surface”of the page rather than deeper “into” the page.

Compared to FIG. 3, the eyes 702 of the user in FIG. 7 are much fartheraway from the display 102 of device 100. As such, the ray tracing andprojection (as represented by dashed lines 714) of graphical userinterface object 107 onto the display 102 places depiction 700 ofgraphical user interface object 107 at a lower position on the display102 in FIG. 7 than the depiction 206 of graphical user interface object107 in FIG. 3. This is consistent with producing the effect of the userlooking through a “window” into the virtual 3D operating systemenvironment 300. Monitoring the position of device 100 in addition tothe position of the eyes 702 of the user can provide for a compellingand realistic virtual experience. It should be noted, however, thatmonitoring the device's Frenet frame changes over time alone (i.e., notin conjunction with the position of the user) often provides sufficientinformation for the device's operating system software to createrealistic 3D UI effects. Creating 3D UI effects without knowing theexact position of the user is possible by assuming a typical user'sposition with respect to the device. For example, it is recognized that,ergonomically, there are a small number of positions that are useful forviewing a personal electronic device's display. Small devices aregenerally held closer to the user's eyes, and larger devices aregenerally held farther away from the user's eyes. Additionally, theuser's gaze is generally focused centrally on the display surface of thedevice.

Referring now to FIG. 8, a recessed, “bento box” form factor inside thevirtual 3D environment 812 of a personal electronic device 100 isillustrated, in accordance with one embodiment. As the user moves thedevice 100 around, the shadows 814 depicted on the display 102 couldmove as well, thus increasing the 3D effect perceived by the user.Various other 3D effects could also be employed to increase the 3Deffect perceived by the user. For example, if the objects 802 inside thevirtual 3D environment 812 are rounded, then shines 816 reflecting offthe objects could also change correspondingly. If the objects 802 arediffuse, then their shading could change with device 100's movement.

The recessed, “bento box” form factor inside the virtual 3D environment812 of a personal electronic device 100 may also be constructed withindividual interface objects 802 inside each of a group of sub-boxes800. As shown in FIG. 8, there are three sub-boxes 800 a/800 b/800 c.Each sub-box 800 may have a set of side walls 818 that form the bordersof the recessed sub-box 800. As a user reorients device 100 to “lookinside” each sub-box 800, a “spotlight” illustrated by dashed lines 804and dashed circle 806 can highlight the sub-box into which the user'sview is currently directed. In some embodiments, a decision as towhether the user is looking into a particular sub-box may be based on,for example, the fact that all walls 818 of that sub-box 800 areforward-facing to the user, and thus visible.

As illustrated in FIG. 8, the user 808 a is hovering above device 100and looking down into the display 102 of the device 100, as representedby sightline 810 a and the fact that no side surfaces of the graphicaluser interface objects 802/803/805/807 are visible on the currentlyrendered display 102. For user 808 a, the spotlight effect 804/806 isapplied to the display 102 in the area of sub-box 800 a and serves tohighlight the representations of graphical user interface objects 802and 803. A spotlight 806 may be considered to be a concentrated virtuallight source that is directed at a particular graphical user interfaceobject(s) or area, e.g., sub-box 800 a in FIG. 8, in the virtual 3Doperating system environment. In the graphics arts, displayed objectcolor may be computed by multiplying the light source's color andintensity by the display object's color. A spotlight may be representedas a light source where the light intensity falls off as a function ofdistance or angular displacement from the center light vector, which maybe calculated by subtracting the light source's position from thelocation where the light is pointed. This function may be, e.g., afunction characterized by having an exponential falloff.

Users 808 b and 808 c represent alternate positions from which a usercould attempt to view the virtual 3D environment 812. From theperspective of user 808 b looking along sightline 810 b, all walls 818of sub-box 800 b would be forward-facing to the user 808 b, and thus thespotlight effect 804/806 would be applied to the display 102 in the areaof sub-box 800 b and serve to highlight the representations of graphicaluser interface object 805. From the perspective of user 808 c lookingalong sightline 810 c, all walls 818 of sub-box 800 c would beforward-facing to the user 808 c, and thus the spotlight effect 804/806would be applied to the display 102 in the area of sub-box 800 c andserve to highlight the representations of graphical user interfaceobject 807. It is understood that other variants to the recessed, “bentobox” form factor fall within the scope of the teachings of thisdisclosure and could likewise be implemented to take advantage oftracking a 3D frame of reference of the device and/or the position ofthe user's eyes and/or head to create a more realistic virtual 3Dexperience for the user.

Referring now to FIG. 9, a point of contact 902 with the touchscreendisplay 102 of a personal electronic device 101 that is ray traced 904into a virtual 3D operating system environment 300 is illustrated, inaccordance with one embodiment. By ray tracing the location of the touchpoint of contact 902 on the device's display 102 into the virtual 3Doperating system environment 300 (as described above in reference toFIG. 3) and intersecting the region of the touch point in the virtual 3Denvironment 906 with whatever object or objects it hits, the user canperceive that he or she is interacting with a 3D environment via a 2Dtouchscreen interface, providing a richer and more realistic userexperience. As illustrated in FIG. 9, the location in which the user 900has touched the touchscreen display 102 corresponds to touch point 906in virtual 3D environment 300 and intersects with graphical userinterface object 107. Thus, a touch by user 900 creating point ofcontact 902 could result in applying a motion or effect to object 107 inthe virtual 3D operating system environment 300 that would cause object107 in the virtual 3D operating system environment 300 to behavesimilarly to how it would behave if the device was operating in atraditional “2D” mode. For example, the techniques disclosed hereinwould make it possible to simulate resizing, depressing, toggling,dragging, pushing, pulling, collision effects, and other physicalmanifestations of reality within the virtual 3D operating systemenvironment 300 in a more compelling and realistic manner. In the caseof simulating a “depress” 3D UI effect, e.g., the depressing of agraphical user interface object such as a button or icon 107, the effectmay be implemented in response to the detected position 902 of theuser's finger 900. This can be done because, once the touch location 902of the user's finger 900 is located in the virtual 3D environment 300,e.g., represented by dashed circle 906 in FIG. 9, it may be found thatthe touch location intersects the plane of some graphical user interfaceobject, e.g., the front plane of icon 107, and any touch movement may betranslated into that plane or interpreted in the context of thatparticular graphical user interface object's potential degrees offreedom of movement. For example, some objects may only be able to be“depressed” inwardly, whereas others may be free to move in any numberof directions.

In another embodiment, a shadow or other indicator 902 of the user 900'sfingertip may be displayed in the appropriate place in the 2D renderingof the virtual 3D operating system environment 300 depicted on display102. Information about the position of the user 900's fingertip can beobtained from contact information reported from the touchscreen or fromnear-field sensing techniques, each of which is known in the art. Inthis way, the user of device 100 can actually feel like he or she is“reaching into” the virtual 3D environment 300. Using near-field sensingtechniques, a finger's position in the “real world” may be translatedinto the finger's position in the virtual 3D operating systemenvironment by reinterpreting the distance of the finger from thedevice's display as a distance of the finger from the relevant graphicaluser interface object in the virtual 3D operating system environment,even when the relevant graphical user interface object is at some“distance” into the virtual 3D world within the display's “window.” Forexample, utilizing the techniques disclosed herein, if a user's fingerwere sensed to be one centimeter from display 102, the relevantindication of the location of the user's touch point in the virtual 3Doperating system environment, e.g., dashed circle 906 in FIG. 9, maycast a shadow or display some other visual indicator “in front of” therelevant graphical user interface object, thus providing an indicationto the user that they are not yet interacting with the relevantgraphical user interface object, but if the user were to move theirfinger closer to the display's “window,” i.e., touch surface display102, they may be able to interact with the desired graphical userinterface object. The use of a visual indicator of the user's desiredtouch location, e.g., the “shadow offset,” is not only an enhanced 3D UIeffect. Rather, because knowledge of the 3D frame of reference of thedevice allows for better accounting of touchscreen parallax problems,i.e., the misregistration between the touch point and the intended touchlocation being displayed, the techniques described herein with respectto FIG. 9 may also provide the user of the device with a betterrepresentation of what interactions he or she would be experiencing ifthe graphical user interface objects were “real life” objects.

Referring now to FIG. 10, an exemplary gesture 1002 for activating thedisplay 102 of a personal electronic device 100 to operate in a virtual3D operating system environment mode is illustrated, in accordance withone embodiment. As mentioned above, constantly using the device's GPUfor rendering 3D information or ray tracing is computationally expensiveand can be a battery drain. In some embodiments, the device will operatein 2D or 2½D mode by default. Thus, a gesture can be used to “unfreeze”the 2D or 2½D display to operate in 3D mode. One potential “activationgesture” is the so-called “princess wave,” i.e., the wave-motionrotation of the device about its Y-axis 1000. For example, the virtual3D operating system environment mode can be turned on when more thanthree waves of 10-20 degrees 1002 of modulation along one axis 1000occur within a predetermined threshold amount of time, e.g., one second.Position quiescence, e.g., holding the device relatively still for atleast a predetermined threshold amount of time, e.g., two to threeseconds, could be one potential cue to the device 100 to freeze back tothe 2D or 2½D operating system environment mode and restore the displayof objects to their traditional 2D representations. In this way, it isunlikely that the device 100 could be put into 3D mode without explicitcontrol and intent from the user. It is also likely that the devicewould return to the computationally cheaper 2D or 2½D modeautomatically, if left alone for a sufficient amount of time.

In one embodiment, when the appropriate activating gesture 1002 foractivating the display 102 of a personal electronic device is detectedand the virtual 3D operating system environment mode turns on, each ofthe graphical user interface objects on the display of the device may“unfreeze” and turns into a 3D depiction of the object, e.g., adepiction that is nearly identical to the 2D depiction of the object,along with shading, shine reflections 1012, and/or textures indicativeof 3D object appearance. For example, each of the icons 1006 on aspringboard 110, such as is shown on the display 102 of the device 100in FIG. 10, could transform from 2D representations of icons into 3D“Lucite” cubes 1004/1006, i.e., cubes that appear to be made of a clearplastic or glass-like material and that have pictures at the bottom ofthem. (LUCITE® is a registered trademark of Lucite International, Inc.)When the icons are unfrozen into the 3D mode, the pictures at thebottoms of the icon cubes 1004/1006 could refract and distortappropriately, providing a visually distinct impression. It is alsopossible for the cubes to be slightly rounded on their front surfaces.This can be done both to magnify the icons below them and also to catchreflected “light” off their surfaces when the display is reoriented withrespect to the user's gaze.

Referring now to FIG. 11, one embodiment of a process for operating apersonal electronic device in a virtual 3D operating system environmentmode is illustrated in flowchart form. First, the process begins at Step1100. Next, one or more processors or other programmable control deviceswithin the personal electronic device defines a virtual 3D operatingsystem environment (Step 1102). Next, the processor(s) may receive datafrom one or more position sensors disposed within a hand-held device(Step 1104). The processor(s) may then determine a 3D frame of referencefor the device based on the received position data (Step 1106). Next,the processor(s) may receive data from one or more optical sensors,e.g., an image sensor, a proximity sensor, or a video camera disposedwithin a hand-held device (Step 1108). The processor(s) may thendetermine a position of a user of the device's eyes and/or head based onthe received optical data (Step 1110). The optical data may comprise oneor more of: two-dimensional image data, stereoscopic image data,structured light data, depth map data, and Lidar data. Based on the typeof optical data received (and the success or failure of a particulartechnique in a given situation), the processor(s) may determine theposition of the user of the device's eyes and/or head using any numberof techniques. For example: stereo imaging methods; structured lighttechniques (which project a pattern of light on the subject and look atthe deformation of the pattern on the subject); “Time of Flight” orother Lidar techniques (which uses laser light to probe the subject andfind the distance of a surface by timing the round-trip time of a pulseof light); and/or any other form of 3D range determining techniques maybe used to estimate the distance to the user's eyes and/or head. In somesituations, two-dimensional image data may be sufficient tosatisfactorily locate the eyes and/or head of the user. In othersituations, however, due to, e.g., poor lighting conditions or poorimage quality, other distance measurement techniques, such as thosetechniques utilizing infrared light, structured light, laser beams mayprove more successful. In another embodiment, Step 1110 may be omitted,and a fixed, assumed position for the user may be used by theprocessor(s). For example, it is recognized that, ergonomically, thereare a small number of positions that are useful for viewing a personalelectronic device's display. Small devices are generally held closer tothe user's eyes, and larger devices are generally held farther away fromthe user's eyes. Additionally, the user's gaze is generally focusedcentrally on the display surface of the device. Finally, theprocessor(s) may then project, i.e., render, a visual representation ofthe virtual 3D operating system environment onto a graphical userinterface display of the device based on the determined 3D frame ofreference and the determined position of the user of the device's eyesand/or head (Step 1112). As discussed in detail above, this process canprovide for a distinct and more realistic presentation of a virtual 3Doperating system environment onto the display of a user device, e.g., ahand-held, personal electronic device such as a mobile phone.

Referring now to FIG. 12, one embodiment of a process for toggling apersonal electronic device between operating in a virtual 3D operatingsystem environment mode and a non-virtual 3D operating systemenvironment mode, e.g., a 2D mode, is illustrated in flowchart form.First, the process begins at Step 1200. Next, one or more processors orother programmable control devices within the personal electronic devicemay display a 2D representation of an operating system environment on agraphical user interface display of a device (Step 1202). Then, theprocessor(s) detects whether it has received a gesture at the deviceindicative of a desire of the user to enter into a virtual 3D displaymode, e.g., a one-axis sine wave modulation of the device of asufficient number of degrees and within a sufficiently short amount oftime (Step 1204). If it has not received such a gesture, the processreturns to Step 1202 and continues to display a 2D representation of theoperating system environment. If, instead, the processor(s) has receivedsuch a gesture, the process proceeds to Step 1206, which indicates tothe processor(s) to perform the process described in FIG. 11, i.e., aprocess for operating the personal electronic device in a virtual 3Doperating system environment mode. While operating in the virtual 3Doperating system environment mode, the processor(s) continues to“listen” for gestures at the device indicative of a desire of the userto return to a 2D display mode, e.g., quiescence (less than somethreshold of motion for a threshold amount of time (Step 1208). If ithas not received such a gesture, the process returns to Step 1206 andcontinues to display a virtual 3D representation of the operating systemenvironment according to the process described in FIG. 11. If, instead,the processor(s) detects that it has received such a gesture indicativeof a desire of the user to return to a 2D display mode, the processreturns to Step 1202, i.e., a process for operating the personalelectronic device in a 2D display mode.

Referring now to FIG. 13, one embodiment of a process for projectingspotlights indicative of the position of a user's eyes into a virtual 3Doperating system environment of a personal electronic device isillustrated in flowchart form. First, the process begins at Step 1300.Next, one or more processors or other programmable control deviceswithin the personal electronic device defines a virtual 3D operatingsystem environment (Step 1302). Next, the processor(s) may project,i.e., render, a visual representation of the virtual 3D operating systemenvironment onto a graphical user interface display of the device (Step1304). Next, the processor(s) may receive data from one or more opticalsensors, e.g., an image sensor, a proximity sensor, or a video camera,disposed within a hand-held device (Step 1306). The processor(s) maythen determine a position of a user of the device's eyes and/or headbased on the received optical data (Step 1308). Finally, theprocessor(s) may then project, i.e., render, a visual representation of“spotlights” into the virtual 3D operating system environment based onthe determined position of the user of the device's eyes and/or head(Step 1310), at which the point the process may return to Step 1306,wherein the processor(s) may receive a stream of data from the one ormore optical sensors disposed within the hand-held device to allow thedevice's display to be updated accordingly as the user's eyes continueto move to and focus on different areas of the display.

Referring now to FIG. 14, one embodiment of a process for implementinggraphical user interface effects on the display of a personal electronicdevice based on ambient light sources detected in the environment of thedevice and/or the relative position of the device is illustrated inflowchart form. First, the process begins at Step 1400. Next, one ormore processors or other programmable control devices within thepersonal electronic device defines a virtual “2½D” operating systemenvironment for a device, that is, an enhanced 2D representation of anoperating system environment possessing certain additional visual cueson icons, toolbars, windows, etc., such as shading and/or reflections,which additional cues are used to further heighten the “3D appearance”of the 2D icons. In some embodiments, the techniques described withrespect to FIG. 14 may also be applied to a virtual 3D operating systemenvironment (Step 1402). Next, the processor(s) may project, i.e.,render, a visual representation of the virtual 2½D operating systemenvironment onto a graphical user interface display of the device (Step1404). Next, the processor(s) may receive data from one or more opticalsensors, e.g., an image sensor, a proximity sensor, or a video camera,disposed within a hand-held device (Step 1406). The processor(s) maythen determine a position of one or more ambient light sources based onthe received optical data (Step 1408). Next, the processor(s) mayreceive data from one or more position sensors disposed within ahand-held device (Step 1410). The processor(s) may then determine a 3Dframe of reference for the device based on the received position data(Step 1412). Finally, the processor(s) may apply UI effects (e.g.,shine, shading) to objects in the virtual 2½D operating systemenvironment based on the determined position of the ambient lightsources and/or the determined 3D frame of reference for the device (Step1414). As discussed above in reference to FIG. 2, various graphicallayers, such as shadows and shine maps may be dynamically re-positionedwithout being re-rendered based on the position of the device and/or thelocation of an ambient light source. Further, adjustments of scale andposition can also be made to various graphical user interface objects inorder to make the display appear to be even more “3D-like” than italready appears.

Referring now to FIG. 15, a simplified functional block diagram of arepresentative personal electronic device 1500 according to anillustrative embodiment, e.g., a mobile phone possessing a camera devicesuch as optical sensor 112, is shown. The personal electronic device1500 may include a processor(s) 1516, storage device 1514, userinterface 1518, display 1520, coder/decoder (CODEC) 1502, bus 1522,memory 1512, communications circuitry 1510, a speaker or transducer1504, a microphone 1506, position sensors 1524, proximity sensor 1526,and an optical sensor with associated hardware 1508. Processor 1516 maybe any suitable programmable control device, including a GPU, and maycontrol the operation of many functions, such as the 3D user interfaceeffects discussed above, as well as other functions performed bypersonal electronic device 1500. Processor 1516 may drive display 1520and may receive user inputs from the user interface 1518. In someembodiments, device 1500 may possess one or more processors forperforming different processing duties.

Storage device 1514 may store media (e.g., photo and video files),software (e.g., for implementing various functions on device 1500),preference information (e.g., media playback preferences), personalinformation, and any other suitable data. Storage device 1514 mayinclude one more storage mediums, including for example, a hard-drive,permanent memory such as ROM, semi-permanent memory such as RAM, orcache.

Memory 1512 may include one or more different types of memory which maybe used for performing device functions. For example, memory 1512 mayinclude cache, ROM, and/or RAM. Bus 1522 may provide a data transferpath for transferring data to, from, or between at least storage device1514, memory 1512, and processor 1516. CODEC 1502 may be included toconvert digital audio signals into analog signals for driving thespeaker 1504 to produce sound including voice, music, and other likeaudio. The CODEC 1502 may also convert audio inputs from the microphone1506 into digital audio signals for storage in memory 1512 or storagedevice 1514. The CODEC 1502 may include a video CODEC for processingdigital and/or analog video signals.

User interface 1518 may allow a user to interact with the personalelectronic device 1500. For example, the user input device 1518 can takea variety of forms, such as a button, keypad, dial, a click wheel, or atouchscreen. Communications circuitry 1510 may include circuitry forwireless communication (e.g., short-range and/or long rangecommunication). For example, the wireless communication circuitry may beWi-Fi® enabling circuitry that permits wireless communication accordingto one of the 802.11 standards. (Wi-Fi® is a registered trademark of theWi-Fi Alliance.) Other wireless network protocols standards could alsobe used, either as an alternative to the identified protocols or inaddition to the identified protocols. Other network standards mayinclude BLUETOOTH®, the Global System for Mobile Communications (GSM®),and code division multiple access (CDMA) based wireless protocols.(BLUETOOTH® is a registered trademark of Bluetooth SIG, Inc., and GSM®is a registered trademark of GSM Association.) Communications circuitry1510 may also include circuitry that enables device 1500 to beelectrically coupled to another device (e.g., a computer or an accessorydevice) and communicate with that other device.

In one embodiment, the personal electronic device 1500 may be a personalelectronic device capable of processing and displaying media such asaudio and video. For example, the personal electronic device 1500 may bea media device such as media player, e.g., a mobile phone, an MP3player, a game player, a remote controller, a portable communicationdevice, a remote ordering interface, an audio tour player, or othersuitable personal device. The personal electronic device 1500 may bebattery-operated and highly portable so as to allow a user to listen tomusic, play games or video, record video, stream video, take pictures,communicate with others, interact with a virtual operating systemenvironment, and/or control other devices. In addition, the personalelectronic device 1500 may be sized such that it fits relatively easilyinto a pocket or hand of the user. By being hand-held, the personalcomputing device 1500 may be relatively small and easily handled andutilized by its user and thus may be taken practically anywhere the usertravels.

As discussed previously, the relatively small form factor of certaintypes of personal electronic devices 1500, e.g., personal media devices,enables a user to easily manipulate the device's position, orientation,and movement. Accordingly, the personal electronic device 1500 mayprovide for improved techniques of sensing such changes in position,orientation, and movement to enable a user to interface with or controlthe device 1500 by affecting such changes. For example, position sensors1524 may comprise compasses, accelerometers, gyrometers, or GPS units.Further, the device 1500 may include a vibration source, under thecontrol of processor 1516, for example, to facilitate sending motion,vibration, and/or movement information to a user related to an operationof the device 1500. The personal electronic device 1500 may include oneor more optical sensors and associated hardware 1508 that enables thedevice 1500 to capture an image or series of images, i.e., video,continuously, periodically, at select times, and/or under selectconditions. The personal electronic device 1500 may also includeproximity sensors 1526 that enable the device 1500 to characterize andidentify light sources in the real world environment surrounding thedevice 1500 and make determinations of whether, e.g., a user or thefinger of a user is in close proximity to the display 1520 of device1500.

The foregoing description is not intended to limit or restrict the scopeor applicability of the inventive concepts conceived of by theApplicants. As one example, although the present disclosure focused on3D user interface effects for a virtual operating system environment; itwill be appreciated that the teachings of the present disclosure can beapplied to other contexts, e.g.: digital photography, digitalvideography, television, video gaming, biometrics, or surveillance. Inexchange for disclosing the inventive concepts contained herein, theApplicants desire all patent rights afforded by the appended claims.Therefore, it is intended that the appended claims include allmodifications and alterations to the full extent that they come withinthe scope of the following claims or the equivalents thereof.

What claimed is:
 1. A graphical user interface method, comprising:receiving positional data from one or more position sensors disposedwithin a device; determining a 3D frame of reference for the devicebased at least in part on the received positional data; displaying aplurality of graphical user interface objects on a display of the deviceat a first orientation with respect to the 3D frame of reference;receiving optical data from one or more optical sensors disposed withinthe device; receiving non-optical data from one or more non-opticalsensors; determining a position of a user of the device's head based, atleast in part, on the received optical data and the received non-opticaldata; determining a position of a graphical layer with respect to atleast one graphical user interface object based at least in part on thedetermined 3D frame of reference and the determined position of the userof the device's head, the graphical layer creating a lighting effectwith respect to the at least one graphical user interface object;generating a virtual 3D depiction of the at least one graphical userinterface object on the display of the device with the graphical layerpositioned with respect to the at least one graphical user interfaceobject; monitoring the position of the device and the position of theuser of the device's head; and adjusting the generated virtual 3Ddepiction of the at least one graphical user interface object on thedisplay of the device, in response to movement of the device or theposition of the user of the device's head, to depict at least one of: atop surface of the at least one graphical user interface object, a sidesurface of the at least one graphical user interface object, or behindthe at least one graphical user interface object, wherein the at leastone graphical user interface object is represented in a virtual 3Doperating system environment.
 2. The graphical user interface method ofclaim 1, wherein the act of generating further comprises applying anappropriate perspective transformation to the virtual 3D depiction ofthe at least one graphic& user interface object on the display of thedevice, wherein the transformation is based, at least in part, on thedetermined 3D frame of reference, the optical data, and the non-opticaldata.
 3. The graphical user interface method of claim 1, furthercomprising the act of receiving data from a touchscreen interface of thedevice, wherein the data received from the touchscreen interface isindicative of one or more locations in the virtual 3D operating systemenvironment with which the user desires to interact.
 4. The graphicaluser interface method of claim 1, wherein the act of adjusting thegenerated virtual 3D depiction of the at least one graphical userinterface object further comprises applying an appropriate perspectivetransformation to the virtual 3D depiction of the at least one graphicaluser interface object on the display of the device, wherein thetransformation is based at least in part on a current determined 3Dframe of reference for the device and the position of the user of thedevice's head.
 5. The graphical user interface method of claim 1,wherein the act of generating further comprises generating a virtuallight source for the virtual 3D operating system environment.
 6. Thegraphical user interface method of claim 1, wherein the act ofgenerating further comprises: ray tracing from the user's eyes throughthe display of the device and into the virtual 3D operating systemenvironment.
 7. The graphical user interface method of claim 1, whereinthe one or more non-optical sensors comprises one or more of: a sensorworn on the user of the device's head; a sensor on headgear worn by theuser of the device; and a sensor that is part of a head-mounted displayworn by the user of the device.
 8. A graphical user interface,comprising: a viewing surface; a virtual 3D operating systemenvironment; one or more graphical user interface objects; and agraphical layer positioned with respect to the one or more graphicaluser interface objects, the graphical layer creating a lighting effectwith respect to the at least one graphical user interface object;wherein the one or more graphical user interface objects and thegraphical layer are represented in the virtual 3D operating systemenvironment and depicted on the viewing surface, and wherein thedepiction of the one or more graphical user interface objects on theviewing surface and the graphical layer positioned with respect to theone or more graphical user interface objects is determined at least inpart by a determined 3D frame of reference of the viewing surface withrespect to a position of a head of a user of the viewing surface.
 9. Thegraphical user interface of claim 8, wherein the viewing surfacecomprises a touchscreen interface.
 10. The graphical user interface ofclaim 9, wherein data received from the touchscreen interface isindicative of one or more locations in the virtual 3D operating systemenvironment with which the user of the viewing surface desires tointeract.
 11. The graphical user interface of claim 10, furthercomprising one or more visual indicators on the viewing surface, whereinthe one or more visual indicators are indicative of the one or morelocations in the virtual 3D operating system environment with which theuser of the viewing surface desires to interact.
 12. The graphical userinterface of claim 11, wherein the locations of the one or more visualindicators on the viewing surface are corrected for parallax problems,wherein a determination of the corrected locations is based at least inpart on the determined 3D frame of reference of the viewing surface. 13.The graphical user interface of claim 8, wherein the depiction of the atleast one graphical user interface object on the viewing surfacecomprises the application of one or more of the following visual effectsto the at least one graphical user interface object: shading, shine,blur, and alpha masking.
 14. The graphical user interface of claim 8,wherein the position of the head of the user of the viewing surface isdetermined based, at least in part, on information received from one ormore of: a sensor worn on the user's head; a sensor on headgear worn bythe user; and a sensor that is part of a head-mounted display worn bythe user.
 15. An apparatus, comprising: a display; one or more opticalsensors; one or more positional sensors; a memory; and one or moreprogrammable control devices communicatively coupled to the display, theone or more optical sensors, the one or more positional sensors, and thememory, wherein the memory includes instructions for causing the one ormore programmable control devices to: receive positional data from theone or more positional sensors; determine a 3D frame of reference forthe apparatus based at least in part on the received positional data;display a plurality of graphical user interface objects on the displayat a first orientation with respect to the 3D frame of reference;receive optical data from the one or more optical sensors; receivenon-optical data from one or more non-optical sensors; determine aposition of a user's head based, at least in part, on the optical dataand the non-optical data; determine a position of a graphical layer withrespect to the at least one graphical user interface object based atleast in part on the determined 3D frame of reference and the determinedposition of the user's head, the graphical layer creating a lightingeffect with respect to the at least one graphical user interface object;render a virtual 3D depiction of at least one graphical user interfaceobject and the graphical layer positioned with respect to the at leastone graphical user interface object on the display; monitor the positionof the apparatus and the position of the user's head; and reposition thegraphical layer with respect to the at least one graphical userinterface object in response to a change in the 3D frame of referencebased on the positional data or the position of the user's head todepict at least one of: a top surface of the at least one graphical userinterface object, a side surface of the at least one graphical userinterface object, or behind the at least one graphical user interfaceobject; wherein the at least one graphical user interface object isrepresented in a virtual 3D operating system environment.
 16. Theapparatus of claim 15, wherein the one or more positional sensorscomprise one or more of the following: a compass, an accelerometer, aGPS module, and a gyrometer.
 17. The apparatus of claim 15, wherein theone or more optical sensors comprise one or more of the following: aproximity sensor, an image sensor, and a video camera.
 18. The apparatusof claim 15, wherein the one or more non-optical sensors comprises oneor more of: a sensor worn on the user's head; a sensor on headgear wornby the user; and a sensor that is part of a head-mounted display worn bythe user.
 19. The apparatus of claim 15, wherein the instructions torender further comprise instructions to apply an appropriate perspectivetransformation to the virtual 3D depiction of the at least one graphicaluser interface object.
 20. The apparatus of claim 19, wherein theperspective transformation is based, at least in part, on the determined3D frame of reference, the optical data, and the non-optical data.